Methods and systems for detecting and processing speech signals

ABSTRACT

Provided are methods, systems, and apparatuses for detecting, processing, and responding to audio signals, including speech signals, within a designated area or space. A platform for multiple media devices connected via a network is configured to process speech, such as voice commands, detected at the media devices, and respond to the detected speech by causing the media devices to simultaneously perform one or more requested actions. The platform is capable of scoring the quality of a speech request, handling speech requests from multiple end points of the platform using a centralized processing approach, a de-centralized processing approach, or a combination thereof, and also manipulating partial processing of speech requests from multiple end points into a coherent whole when necessary.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation of U.S. patent application Ser. No.15/052,426, entitled “Method and System for Detecting and ProcessingSpeech Signals,” filed Feb. 24, 2016, of which is hereby incorporated byreference in its entirety.

BACKGROUND

Media data (e.g., audio/video content) is sometimes shared betweenmultiple modules on a network. To get the most out of such media sharingarrangements, it is desirous to have a platform that is capable ofprocessing such media data from the multiple modules simultaneously.

SUMMARY

This Summary introduces a selection of concepts in a simplified form inorder to provide a basic understanding of some aspects of the presentdisclosure. This Summary is not an extensive overview of the disclosure,and is not intended to identify key or critical elements of thedisclosure or to delineate the scope of the disclosure. This Summarymerely presents some of the concepts of the disclosure as a prelude tothe Detailed Description provided below.

The present disclosure generally relates to methods and systems forprocessing audio signals. More specifically, aspects of the presentdisclosure relate to detecting and processing speech signals frommultiple end points simultaneously.

One embodiment of the present disclosure relates to a method comprising:detecting, at one or more data modules in a group of data modules incommunication with one another over a network, an activation command;computing, for each of the one or more data modules, a score for thedetected activation command; receiving audio data from each detectingdata module having a computed score above a threshold; sending a requestto a server in communication with the group of data modules over thenetwork, wherein the request includes the audio data received from eachof the detecting data modules having a computed score above thethreshold; receiving from the server, in response to the sent request,audio data associated with a requested action; and communicating therequested action to each of the data modules in the group of datamodules.

In another embodiment, the method further comprises: combining the audiodata received from each of the detecting data modules having a computedscore above the threshold; and generating the request to the serverbased on the combined audio data.

In another embodiment, the method further comprises, in response todetecting the activation command, muting a loudspeaker of each datamodule in the group.

In yet another embodiment, the method further comprises activating amicrophone of each detecting data module having a computed score abovethe threshold.

In still another embodiment, the method further comprises causing eachof the data modules in the group to playout an audible confirmation ofthe requested action communicated to each of the data modules.

Another embodiment of the present disclosure relates to a systemcomprising a group of data modules in communication with one anotherover a network, where each of the data modules is configured to: inresponse to detecting an activation command, compute a score for thedetected activation command; determine whether the computed score forthe activation command is higher than a threshold number of computedscores for the activation command received from other detecting datamodules in the group; in response to determining that the computed scorefor the activation command is higher than the threshold number ofcomputed scores received from the other detecting data modules, sendaudio data recorded by the data module to a server in communication withthe group of data modules over the network; receive from the server, inresponse to the sent audio data, a requested action; determine aconfidence level for the requested action received from the server; andperform the requested action based on a determination that theconfidence level determined by the data module is higher than confidencelevels determined by a threshold number of other data modules thatreceived the requested action from the server.

In another embodiment, each of the data modules in the system isconfigured to, in response to computing the score for the detectedactivation command, send the computed score to each of the other datamodules in the group.

In another embodiment, each of the data modules in the system isconfigured to receive, from other detecting data modules in the group,scores for the activation command computed by the other detecting datamodules.

In another embodiment, each of the data modules in the system isconfigured to broadcast the determined confidence level to the otherdata modules in the group that received the requested action from theserver.

In another embodiment, each of the data modules in the system isconfigured to: compare the confidence level determined by the datamodule to confidence levels broadcasted by the other data modules in thegroup that received the requested action from the server; and determine,based on the comparison, that the confidence level determined by thedata module is higher than the confidence levels determined by thethreshold number of other data modules that received the requestedaction from the server.

In yet another embodiment, each of the data modules in the system isconfigured to, in response to determining that the confidence leveldetermined by the data module is higher than the confidence levelsdetermined by the threshold number of other data modules, playout anaudible confirmation of the request action received from the server.

In still another embodiment, each of the data modules in the system isconfigured to compute a score for the detected activation command basedon one or more of the following: a power of a signal received at thedata module for the activation command; a determined location of asource of the activation command relative to the data module; andwhether the detected activation command corresponds to a previouslystored activation command.

In one or more other embodiments, the methods and systems describedherein may optionally include one or more of the following additionalfeatures: the computed score for the activation command detected at adata module is based on one or more of a power of a signal received atthe data module for the activation command, a determined location of asource of the activation command relative to the data module, andwhether the detected activation command corresponds to a previouslystored activation command; the audio data received from each detectingdata module having a computed score above the threshold includes speechdata captured and recorded by the data module; the speech data capturedand recorded by the data module is associated with a speech commandgenerated by a user; the speech data captured by each data module withan activated microphone is associated with a portion of a speech commandgenerated by a user; the audio data recorded by the data module includesspeech data recorded by the data module; the speech data recorded by thedata module is associated with a speech command generated by a user; theconfidence level for the requested action is determined based on anaudio quality measurement for the audio data recorded by the data moduleand sent to the server, and/or the requested action received from theserver is based on audio data recorded by a plurality of the otherdetecting data modules having computed scores higher than the thresholdnumber of computed scores.

It should be noted that embodiments of some or all of the processor andmemory systems disclosed herein may also be configured to perform someor all of the method embodiments disclosed above. In addition,embodiments of some or all of the methods disclosed above may also berepresented as instructions embodied on transitory or non-transitoryprocessor-readable storage media such as optical or magnetic memory orrepresented as a propagated signal provided to a processor or dataprocessing device via a communication network such as an Internet ortelephone connection.

Further scope of applicability of the methods and systems of the presentdisclosure will become apparent from the Detailed Description givenbelow. However, it should be understood that the Detailed Descriptionand specific examples, while indicating embodiments of the methods andsystems, are given by way of illustration only, since various changesand modifications within the spirit and scope of the concepts disclosedherein will become apparent to those skilled in the art from thisDetailed Description.

BRIEF DESCRIPTION OF DRAWINGS

These and other objects, features, and characteristics of the presentdisclosure will become more apparent to those skilled in the art from astudy of the following Detailed Description in conjunction with theappended claims and drawings, all of which form a part of thisspecification. In the drawings:

FIG. 1 is a block diagram illustrating an example content managementsystem and surrounding network environment according to one or moreembodiments described herein.

FIG. 2 is a flowchart illustrating an example method for detecting,processing, and responding to speech signals from multiple end pointsaccording to one or more embodiments described herein.

FIG. 3 is a block diagram illustrating example components and data flowsfor detecting a speech command in a multi-device content managementsystem according to one or more embodiments described herein.

FIG. 4 is a block diagram illustrating example components and data flowsfor assessing quality of a detected speech command in a multi-devicecontent management system according to one or more embodiments describedherein.

FIG. 5 is a block diagram illustrating example components and data flowsfor activating a device based on a detected speech command in amulti-device content management system according to one or moreembodiments described herein.

FIG. 6 is a block diagram illustrating example components and data flowsfor processing a detected speech command in a multi-device contentmanagement system according to one or more embodiments described herein.

FIG. 7 is a block diagram illustrating example components and data flowsfor responding to a speech command in a multi-device content managementsystem according to one or more embodiments described herein.

FIG. 8 is a block diagram illustrating an example computing devicearranged for detecting and processing speech signals from multiple endpoints simultaneously according to one or more embodiments describedherein.

The headings provided herein are for convenience only and do notnecessarily affect the scope or meaning of what is claimed in thepresent disclosure.

In the drawings, the same reference numerals and any acronyms identifyelements or acts with the same or similar structure or functionality forease of understanding and convenience. The drawings will be described indetail in the course of the following Detailed Description.

DETAILED DESCRIPTION

Various examples and embodiments of the methods and systems of thepresent disclosure will now be described. The following descriptionprovides specific details for a thorough understanding and enablingdescription of these examples. One skilled in the relevant art willunderstand, however, that one or more embodiments described herein maybe practiced without many of these details. Likewise, one skilled in therelevant art will also understand that one or more embodiments of thepresent disclosure can include other features not described in detailherein. Additionally, some well-known structures or functions may not beshown or described in detail below, so as to avoid unnecessarilyobscuring the relevant description.

Embodiments of the present disclosure relate to methods, systems, andapparatuses for detecting, processing, and responding to audio (e.g.,speech) within an area or space (e.g., a room). For example, inaccordance with at least one embodiment, a platform for multiple mediadevices connected via a network may be configured to process speech(e.g., voice commands) detected at the media devices, and respond to thedetected speech by causing the media devices to simultaneously performone or more requested actions.

As will be described in greater detail below, the methods and systems ofthe present disclosure use a distributive approach for handling voicecommands by considering input from multiple end points of the platform.Such end points may be, for example, independent data modules (e.g.,media and/or audio devices such as, for example, loudspeakers) connectedto one another via a wired or wireless network (e.g., Wi-Fi, Ethernet,etc.).

The methods and systems described herein utilize a flexible architecturein which each data module (e.g., loudspeaker) plays a unique role (e.g.,has particular responsibilities, privileges, and/or capabilities) indetecting, processing, and responding to speech commands (e.g.,generated by a user). The flexibility of the architecture is partlybased on the ability of the data modules to dynamically switch betweendifferent roles (e.g., operating roles) while the system is in activeoperation.

Among numerous other advantages, features, and functionalities that willbe described in greater detail herein, the methods and systems of thepresent disclosure are capable of scoring the quality of a speechrequest (e.g., voice command, speech command, etc.), handling speechrequests from multiple end points using a centralized processingapproach, a de-centralized processing approach, or a combinationthereof, and also manipulating partial processing of speech requestsfrom multiple end points into a coherent whole when necessary.

For example, in a scenario involving multiple data modules (e.g.,loudspeakers), where each data module has a set of microphones (e.g.,microphone array), each data module may compute (e.g., determine) ascore for audio data (e.g., speech command, activation command, etc.) itrecords. In the following description, the score computed by a datamodule may be referred to as a “Hot Word” score for the data module. Thecomputed Hot Word scores may then be used by the system to evaluatewhich of the data modules received the best signal. In accordance withone or more embodiments, the Hot Word score computed by each of the datamodules may be based on, for example, one or more of the following:

(i) Power of the signal. For example, the power of the signal receivedat the data module for the speech command may be compared to the powerof the signal received prior to the speech command.

(ii) Score of a Hot Word recognizer/detector module (which, for example,might be based on or utilize neural network concepts). For example, inaccordance with at least one embodiment, the audio data received orrecorded at a given data module may be fed to a Hot Word detector, whichmay be configured to determine whether the audio data corresponds to aknown (e.g., stored) Hot Word. For example, the Hot Word detector mayutilize a neural network (NN) or a deep neural network (DNN), whichtakes features of the input audio data and determines (e.g., identifies,assesses, evaluates, etc.) whether there are any occurrences of a HotWord. If a Hot Word is found to be present in the audio data, then thedetector may, for example, set a flag. In accordance with at least oneembodiment, the Hot Word detector may be configured to generate a scorefor any detection of a Hot Word that is made by the detector. The scoremay, for example, reflect a confidence of the NN or DNN with regard tothe detection. For example, the higher the score, the more confident thenetwork is that a Hot Word is present in the audio data. In accordancewith one or more embodiments, the output of the DNN may be a likelihood(e.g., probability) of the Hot Word being present in the audio datarecorded at the data module. The determined likelihood may be comparedto a threshold (e.g., a likelihood threshold, which may be predeterminedand/or dynamically adaptable or adjustable based on, for example,network conditions, scores calculated for other nearby data modules,some combination thereof, and the like), and if the determinedlikelihood is at or above the threshold then a flag may be set toindicate the detection of a Hot Word. The threshold may be set so as toachieve or maintain, for example, a target false-detection versusmiss-detection rate. As will be described in greater detail herein, if aHot Word detection confidence is higher for a particular one of the datamodules, it intuitively follows that the module in question will likelyhave a higher chance of correctly recognizing the command query thatfollows the detected Hot Word.

(iii) Location of the user relative to the data module. For example, byusing a localizer (which, for example, may be part of a beamformer, ormay be a standalone module) the angle of the sound source may beobtained. In another example, the angles provided by different datamodules may be triangulated to estimate the position of the user (thisis based on the assumption that the positions of the data modules areknown).

(iv) Additional processing performed on the audio (e.g., combining allmicrophone array outputs using a beamformer, applying noisesuppression/cancellation, gain control, echo suppression/cancellation,etc.)

In accordance with one or more embodiments, the system of the presentdisclosure may be configured to handle speech requests from multiple endpoints (e.g., data modules) using a centralized processing approach, ade-centralized processing approach, or an approach based on acombination thereof. For example, in accordance with at least oneembodiment, audio data (e.g., speech data) may be collected from allrelevant sources (e.g., end points) in the system and the collectedaudio data sent to one centralized processor (e.g., which may be one ofthe data modules in a group of data modules, as will be furtherdescribed below). The centralized processor may determine (e.g.,identify, select, etc.), based on scores associated with the audio datareceived from each of the sources, one or more of the sources thatrecorded the highest quality audio data (e.g., the processor maydetermine the sources that have scores higher than the scores associatedwith a threshold number of other sources). The centralized processor maysend the audio data received from the sources having the highest scoresto a server (e.g., a server external to the system of data modules) forfurther processing. The centralized processor may then receive aresponse from the server and take appropriate action in accordance withthe response.

In accordance with at least one other embodiment, each data module in agroup of data modules may determine its own Hot Word score and broadcastits score to the other data modules in the group. If a data module inthe group determines, based on the broadcasted scores, that the datamodule has one of the best (e.g., highest quality) signals, then thedata module may send/upload its recorded audio data (e.g., speech datarelating to a command from the user) to the server (e.g., the VoiceSearch Back-End, further details of which will be provided below). Uponreceiving a response from the server, the data module may then broadcastits confidence level of the response and wait for similar broadcastsfrom other data modules in the group. If the data module determines thatit has one of the highest confidence levels for the response, the datamodule may act on the response accordingly.

For example, in accordance with at least one embodiment, when a datamodule detects a Hot Word, the data module generates a score for thedetected Hot Word, broadcasts the score to the other data modules in thegroup (e.g., an Ethernet broadcast), and waits for some period of time(which may be a predetermined period of time, a period of time based ona setting that may or may not be adjustable, or the like) to receivesimilar broadcasts from other modules. After the designated period oftime has passed, the data module has access to the scores generated byall of the other data modules in the group that have also detected theHot Word. As such, the data module (as well as each of the otherdetecting data modules in the group) can then determine (e.g., rank) howwell it scored with respect to the other detecting data modules. Forexample, if the data module determines that it has one of the top (e.g.,two, three, etc.) scores for the Hot Word, the data module can decide totake action.

The system may also be capable of performing partial processing ofspeech commands by utilizing portions of audio data received frommultiple data modules. For example, in accordance with one or moreembodiments, the system may capture each part of a sentence spoken bythe user from the “best” loudspeaker for that particular part. Suchpartial processing may be applicable, for example, when a user speaks acommand while moving around within a room. A per-segment-score may becreated for each data module and each word processed independently. Itshould be noted that because the clocks of the data modules in a givengroup are synchronized, the system is able to compare signal-to-noiseratio (SNR) values between speech segments.

In an example application of the methods and systems of the presentdisclosure, users are given the ability to play audio content availablefrom an audio source (e.g., audio content stored on a user device, audiocontent associated with a URL and accessible through the user device,etc.) to any combination of audio devices that share a common wirelessor wired network. For example, in the context of a multi-room house, asystem of speakers may be located in each room (e.g., living room,dining room, bedroom, etc.) of the house, and the speakers forming asystem for a given room may be at various locations throughout the room.In accordance with one or more embodiments described herein, audio willbe played out synchronously across all of the audio devices selected bythe user. It should be understood, however, that the methods and systemsdescribed herein may be applicable to any system that requires timesynchronization of any data type between different modules on a network,and thus the scope of the present disclosure is not in any way limitedby the example application described above.

FIG. 1 is an example content management system 100 in which one or moreembodiments described herein may be implemented. Data Source 110 (e.g.,a content source such as an audio source (e.g., an online streamingmusic or video service, a particular URL, etc.)) may be connected toData Module 115 over a Network 105 (e.g., any kind of network including,for example, Ethernet, wireless LAN, cellular network, etc.). Content(e.g., audio, video, data, mixed media, etc.) obtained from Data Source110 may be played out by Data Module 115 and/or transported by DataModule 115 to one or more of Data Modules 120 a-120 n (where “n” is anarbitrary number) over Network 125 (e.g., a wireless LAN or Ethernet).Similarly, content obtained at Data Modules 120 a-120 n may be playedout by Data Modules 120 a-120 n and/or transported over Network 135 tocorresponding Data Modules 130 a-130 m, Data Modules 140 a-140 p, orsome combination thereof (where “m” and “p” are both arbitrary numbers).It should also be noted that Networks 125 and 135 may be the same ordifferent networks (e.g., different WLANs within a house, one wirelessnetwork and the other a wired network, etc.).

A Control Client 150 may be in communication with Data Module 115 overNetwork 105. In accordance with at least one embodiment, Control Client150 may act as a data source (e.g., Data Source 110) by mirroring localdata from the Control Client to Data Module 115.

In accordance with one or more embodiments, the data modules (e.g., DataModule 115, Data Modules 120 a-120 n, and Data Modules 130 a-130 m) inthe content management system 100 may be divided into groups of datamodules. Each group of data modules may be divided into one or moresystems, which, in turn, may include one or more individual datamodules. In accordance with at least one embodiment, group and systemconfigurations may be set by the user.

Data modules within a group may operate in accordance with differentroles. For example, data modules within a group may be divided intoPlayer Modules, Follower Modules, and Renderer Modules (sometimesreferred to herein simply as “Players,” “Followers,” and “Renderers,”respectively). Example features and functionalities of the Players,Followers, and Renderers will be described in greater detail below. Inaccordance with at least one embodiment, the methods and systems of thepresent disclosure allow for multiple configurations andPlayer/Follower/Renderer combinations, and further allow suchconfigurations and/or combinations to be modified on-the-fly (e.g.,adaptable or adjustable by the user and/or system while the system is inoperation). As is further described below, the resulting configuration(Player/Follower/Renderer) is determined based on the grouping, audiosource/type, network conditions, etc.

The Player acts as “master” or a “leader” of a group of data modules(e.g., Data Module 115 may be the Player in the example group comprisingData Module 115, Data Modules 120 a-120 n, and Data Modules 130 a-130 min the example content management system 100 shown in FIG. 1). Forexample, the Player may fetch (e.g., retrieve, obtain, etc.) the data(e.g., audio) from the source (e.g., Data Source 110) and forward thedata out to the other data modules (e.g., loudspeakers) in the group.The source of the data obtained by the Player may be, for example, anonline audio/video streaming service or website, a portable user device(e.g., cellular telephone, smartphone, personal digital assistant,tablet computer, laptop computer, smart television, etc.), a storagedevice containing memory for storing audio/video data (e.g., astandalone hard drive), and the like. The Player may also be configuredto packetize the data obtained from the source and send raw or codeddata packets over the network to the data modules in the group. Inaccordance with at least one embodiment, the Player may be configured todetermine whether to send raw or coded data packets to the other datamodules in the group based on available bandwidth of the network and/orthe capabilities of each particular data module (e.g., each Follower orRenderer's capabilities). For example, it may be the case that one ormore devices (e.g., loudspeakers) in the system are not capable ofdecoding all codecs. Also, in a scenario involving degraded (e.g.,limited) network conditions (e.g., low bandwidth), it may be difficultfor the Player to send raw data to the other data modules in the group.In such a scenario, the Player may instead send coded data to the othermodules. In other instances, the system may be configured to re-encodethe data following the initial decoding by the Player. However, itshould also be understood that the data originally received by thePlayer is not necessarily coded in all instances (and thus there may notbe an initial decoding performed by the Player).

In addition to the example features and functionalities of the Playerdescribed above, in accordance with one or more embodiments of thepresent disclosure, the Player may also act as a centralized processorin detecting, processing, and responding to speech commands (e.g.,generated by a user). For example, as will be described in greaterdetail below with respect to the example arrangements illustrated inFIGS. 3-7, the Player (or “Group Leader Module”) may be configured toreceive (e.g., retrieve, collect, or otherwise obtain) “Hot Word”command scores from each of the other data modules in the group,determine the data modules with the highest scores, activate or cause toactivate the microphones on the data modules with the highest scores,receive audio data containing a speech command of the user from the datamodules with the activated microphones, and combine the received audiodata into a request that is sent to an external server for processing(e.g., interpretation). In addition, in response to sending the audiodata containing the user's speech command, the Player may receive fromthe server a response containing a requested action corresponding to thespeech command, which the Player may then fan out (e.g., distribute) tothe other data modules in the group so that the requested action isperformed. In accordance with one or more embodiments described herein,the response received at the Player from the server may also includeaudio data corresponding to the requested action, which the Player mayalso fan out to the other data modules in the group. Such audio datamay, for example, be played out by each of the data modules in the groupas an audible confirmation to the user that the user's command wasreceived and is being acted on.

It should also be understood that a Player may also be a Follower and/ora Renderer, depending on the particulars of the group configuration.

The Follower is the head of a local system of data modules (e.g., DataModules 120 a-120 n may be Followers in different systems of datamodules made up of certain Data Modules 130 a-130 m in the examplecontent management system 100 shown in FIG. 1). The Followers mayreceive data from a Player and fan (e.g., forward) out the data to theconnected Renderers in their respective systems. In accordance with oneor more embodiments, the Follower may receive over the network raw orcoded data packets from the Player and send the data packets to theRenderers in the system. The Follower may send the packets to theRenderers in the same format as the packets are received from thePlayer, or the Follower may parse the packets and perform variousoperations (e.g., transcoding, audio processing, etc.) on the receiveddata before re-packeting the data for sending to the connectedRenderers. It should be noted that a Follower may also be a Renderer.

In accordance with at least one embodiment of the present disclosure,the Renderer is the endpoint of the data pipeline in the contentmanagement system (e.g., Data Modules 130 a-130 m in the example contentmanagement system 100 shown in FIG. 1). For example, the Renderer may beconfigured to playout the data received from the Follower that heads itsrespective system. The Renderer may perform additional local processing(e.g., fade in/fade out in the context of audio) on the data receivedfrom the Follower prior to playing out the data.

As described above, one or more of the data modules in the contentmanagement system may be in communication with and/or receive controlcommands from a control client connected to the network (e.g., ControlClient 150 may be in communication with Data Module 115 over Network 105in the example content management system 100 shown in FIG. 1). Thecontrol client is not a physical data module (e.g., not a physicalloudspeaker), but instead may be a device (e.g., cellular telephone,smartphone, personal digital assistant, tablet computer, laptopcomputer, smart television, etc.) that can control and send messages tothe Player. For example, the control client may be used to relay variouscontrol messages (e.g., play, pause, stop, volume updates, etc.) to thePlayer. In accordance with one or more embodiments, the control clientmay also act as a data source (e.g., Data Source 110) for the contentmanagement system, for example, by mirroring local data from the controlclient to the Player. The control client may use the same communicationprotocol as the data modules in the content management system.

It should be understood that the platform, architecture, and system ofthe present disclosure are extremely dynamic. For example, a user of thesystem and/or the system itself may modify the unique roles of the datamodules, the specific data modules targeted for playout, the grouping ofdata modules, the designation of an “active” group of data modules, orsome combination thereof while the system is in active operation.

In accordance with one or more embodiments of the present disclosure,the selection of a group leader (e.g., a Player Module) may be performedusing a system in which each data module advertises its capabilities toa common system service, which then determines roles for each of themodules, including the election of the group leader, based on theadvertised capabilities. For example, the leader selection process maybe based on a unique score computed (e.g., by the common system service)for each of the data modules (e.g., loudspeakers). In accordance with atleast one embodiment, this score may be computed based on one or more ofthe following non-limiting parameters: (i) CPU capabilities; (ii) codecavailability (e.g., a select or limited number of codecs may beimplemented in particular data modules); and (iii) bandwidth/latency.

FIG. 2 illustrates an example process 200 for detecting, processing, andresponding to speech signals (e.g., speech commands) from multiple endpoints. In accordance with one or more embodiments described herein, oneor more of blocks 205-240 in the example process 200 may be performed byone or more of the components in the example content management systemshown in FIG. 1, and described in detail above. For example, one or moreof Control Client 150, Data Module 115, Data Modules 120 a-120 n, andData Modules 130 a-130 m in the example content management system 100may be configured to perform one or more of the operations associatedwith blocks 205-240 in the example process 200 for detecting,processing, and responding to speech commands, further details of whichare provided below.

It should also be noted that, in accordance with one or moreembodiments, the example process 200 for detecting, processing, andresponding to speech commands may be performed without one or more ofblocks 205-240, and/or performed with one or more of blocks 205-240being combined together.

At block 205, a Hot Word command (which may sometimes be referred toherein as an “activation command,” “initialization command,” or thelike) may be generated (e.g., by a user) during audio playback by datamodules in a group of data modules (e.g., a group of data modulescomprising Data Module 115, Data Modules 120 a-120 n, and Data Modules130 a-130 m in the example content management system 100 shown in FIG.1).

At block 210, the data modules in the group that detect the generatedHot Word command (e.g., which may or may not be all of the data modulesin the group) may determine (e.g., compute, calculate, etc.) a score forthe detected command (a “Hot Word” score). For example, in accordancewith at least one embodiment, the Hot Word score that may be determinedby each of the data modules may be based on, for example, one or more ofthe following non-exhaustive and non-limiting factors: (i) power of thesignal (e.g., the power of the signal received at the data module forthe speech command may be compared to the power of the signal receivedprior to the speech command); (ii) score of a Hot Wordrecognizer/detector module (the details of which are described above);(iii) location of the user relative to the data module. For example, byusing the localizer of a beamformer, the angle of the sound source maybe obtained. In another example, the angles provided by different datamodules may be triangulated to estimate the position of the user (thisis based on the assumption that the positions of the data modules areknown); and (iv) additional processing performed on the audio (e.g.,combining all microphone array outputs using a beamformer, applyingnoise suppression/cancellation, gain control, echosuppression/cancellation, etc.).

At block 215, each of the data modules in the group may send itscomputed “Hot Word” score to a group leader data module (e.g., a PlayerModule, as described above). In accordance with one or more embodimentsof the present disclosure, the group leader data module may act as acentralized processor of sorts in that the group leader collects (e.g.,receives) the computed Hot Word scores from the other data modules inthe group.

At block 220, the group leader data module may pause or mute audioplayback by the other data modules in the group and determine (e.g.,identify), based on the computed Hot Word scores received from the datamodules at block 215, those data modules having the highest computed HotWord scores for the Hot Word command generated at block 205. Forexample, the group leader data module may utilize the received Hot Wordscores (at block 215) to rank or order the data modules in the groupaccording to their corresponding scores. The group leader data modulemay then determine the data modules that have one of the top (e.g., two,three, etc.) scores for the Hot Word command generated at block 205. Inanother example, the group leader data module may determine the datamodules that have Hot Word scores higher than the scores of somethreshold number of the detecting data modules.

At block 225, the group leader data module may activate microphone(s) atthe data module(s) in the group determined to have the highest computedscores for the Hot Word command.

At block 230, the data modules with activated microphones (from block225) may record a generated command/request (e.g., a command/requestgenerated by the user) and send audio data containing the recordedcommand/request to the group leader data module.

At block 235, the group leader data module may generate a request basedon the audio data containing the recorded command/request received fromthe data modules with activated microphones (at block 230), and send thegenerated request to an external server for processing (e.g.,interpretation). For example, the group leader data module may generatethe request sent to the external server by combining the audio datareceived from the data modules. In addition, in accordance with one ormore embodiments, the external server may be a back-end server (e.g.,Voice Search Back-End 660 or 760 as shown in the example component anddata flows in FIGS. 6 and 7, respectively) that receives the requestfrom the group leader and is configured to interpret the combined audiodata (e.g., the audio data containing the recorded command/request from,for example, the user).

At block 240, the group leader data module may receive from the external(e.g., back-end) server a response to the request sent by the groupleader data module (e.g., at block 235). The group leader data modulemay process the received response and take appropriate control actionbased on the response, and/or the group leader module may distribute(e.g., fan out, transmit, etc.) the response to the other data modulesin the group so that the requested action is performed. For example, inaccordance with at least one embodiment, the response received at thegroup leader data module at block 240 may contain a requested actioncorresponding to the generated command/request (e.g., speech command)recorded by the data modules with activated microphones (at block 230).In another example, the response received at the group leader datamodule from the server (at block 240) may also include audio datacorresponding to the requested action, which the group leader datamodule may also fan out to the other data modules in the group. Suchaudio data may, for example, be played out by each of the data modulesin the group as an audible confirmation to the user that the user'scommand was received and is being acted on.

It should be noted that, in accordance with one or more embodiments ofthe present disclosure, one or more of the operations associated withblocks 205-240 in the example process 200 for detecting, processing, andresponding to speech commands may optionally be modified and/orsupplemented without loss of any of the functionalities or featuresdescribed above. For example, each data module in the group of datamodules may determine (e.g., calculate, compute, etc.) its own Hot Wordscore and broadcast its score to the other data modules in the group. Ifa data module in the group determines, based on the broadcasted scores,that the data module has one of the best (e.g., highest quality)signals, then the data module may send/upload its recorded audio data(e.g., speech data relating to a command from the user) to the externalserver for processing/interpretation (e.g., to Voice Search Back-End 660or 760 as shown in the example component and data flows in FIGS. 6 and7, respectively, further details of which are provided below). Uponreceiving a response from the server, the data module may then broadcastits confidence level of the response and wait for similar broadcastsfrom other data modules in the group. If the data module determines thatit has one of the highest confidence levels for the response, the datamodule may act on the response accordingly (e.g., perform a requestedaction contained in the response received from the server).

For example, in accordance with at least one embodiment, when a datamodule detects a Hot Word, the data module may generate a score for thedetected Hot Word, broadcast the score to the other data modules in thegroup (e.g., an Ethernet broadcast), and wait for some period of time(which may be, for example, a predetermined period of time, a period oftime based on a setting that may or may not be adjustable, or the like)to receive similar broadcasts from other data modules. After thedesignated period of time has passed, the data module has access to thescores generated by the other data modules in the group that have alsodetected the Hot Word. As such, the data module (as well as each of theother detecting data modules in the group) can then determine (e.g.,rank) how well it scored with respect to the other detecting datamodules. For example, if the data module determines that it has one ofthe top (e.g., two, three, etc.) scores for the Hot Word, the datamodule can decide to take action (e.g., send/upload its recorded audiodata (e.g., speech data relating to a command from the user) to theexternal server for processing/interpretation).

It should also be noted that the system of the present disclosure mayalso be capable of performing partial processing of speech commands byutilizing portions of audio data received from multiple data modules.For example, in accordance with one or more embodiments, the system maycapture each part of a sentence spoken by the user from the “best”loudspeaker for that particular part. Such partial processing may beapplicable, for example, when a user speaks a command while movingaround within a room. A per-segment-score may be created for each datamodule and each word processed independently. It should be noted thatbecause the clocks of the data modules in a given group aresynchronized, the system is able to compare signal-to-noise ratio (SNR)values between speech segments.

FIGS. 3-7 illustrate example components and data flows for variousoperations that may be performed by the content management system 100shown in FIG. 1 (and described in detail above). In accordance with oneor more embodiments of the present disclosure, a Player data module (or“Group Leader Module”) may also act as a centralized processor indetecting, processing, and responding to speech commands (e.g.,generated by a user). For example, the Player (e.g., 315, 415, 515, 615,and 715 in the example arrangements shown in FIGS. 3-7, respectively)may be configured to receive “Hot Word” command scores from each of theother data modules in the group (e.g., data modules 320 a-320 n, 420a-420 n, 520 a-520 n, 620 a-620 n, and 720 a-720 n in the examplearrangements shown in FIGS. 3-7, respectively), determine the datamodules with the highest scores, activate or cause to activate themicrophones on the data modules with the highest scores, receive audiodata containing a speech command of the user (e.g., 370, 470, 570, 670,and 770) from the data modules with the activated microphones, andcombine the received audio data into a request that is sent to anexternal server (e.g., 660 and 760 in FIGS. 6 and 7, respectively) forprocessing (e.g., interpretation).

In addition, in response to sending the audio data containing the user'sspeech command, the Player (e.g., 715 in FIG. 7) may receive from theserver (760) a response containing a requested action corresponding tothe speech command, which the Player may then fan out (e.g., distribute)to the other data modules in the group (720 a-720 n) so that therequested action is performed. In accordance with one or moreembodiments described herein, the response received at the Player fromthe server may also include audio data corresponding to the requestedaction, which the Player may also fan out to the other data modules inthe group. Such audio data may, for example, be played out by each ofthe data modules in the group as an audible confirmation to the userthat the user's command was received and is being acted on.

FIG. 8 is a high-level block diagram of an exemplary computer (800) thatis arranged for detecting, processing, and responding to speech commandsin a multi-device content management system in accordance with one ormore embodiments described herein. In a very basic configuration (801),the computing device (800) typically includes one or more processors(810) and system memory (820). A memory bus (830) can be used forcommunicating between the processor (810) and the system memory (820).

Depending on the desired configuration, the processor (810) can be ofany type including but not limited to a microprocessor (μP), amicrocontroller (μC), a digital signal processor (DSP), or anycombination thereof. The processor (810) can include one more levels ofcaching, such as a level one cache (811) and a level two cache (812), aprocessor core (813), and registers (814). The processor core (813) caninclude an arithmetic logic unit (ALU), a floating point unit (FPU), adigital signal processing core (DSP Core), or any combination thereof. Amemory controller (815) can also be used with the processor (810), or insome implementations the memory controller (815) can be an internal partof the processor (810).

Depending on the desired configuration, the system memory (820) can beof any type including but not limited to volatile memory (such as RAM),non-volatile memory (such as ROM, flash memory, etc.) or any combinationthereof. System memory (820) typically includes an operating system(821), one or more applications (822), and program data (824). Theapplication (822) may include a system for detecting and processingspeech commands (823). In accordance with at least one embodiment of thepresent disclosure, the system for detecting and processing speechcommands (823) is further designed to perform partial processing ofspeech commands by utilizing portions of audio data received frommultiple data modules in a content management system (e.g., Data Module115, Data Modules 120 a-120 n, and/or Data Modules 130 a-130 m in theexample content management system 100 shown in FIG. 1 and described indetail above).

Program Data (824) may include storing instructions that, when executedby the one or more processing devices, implement a system (823) andmethod for detecting and processing speech commands using multiple datamodules operating on a network. Additionally, in accordance with atleast one embodiment, program data (824) may include network, Hot Words,and module data (825), which may relate to various statistics routinelycollected from the local network on which the system (823) is operating,certain voice/speech commands that activate scoring an processingoperations, as well as one or more characteristics of data modulesincluded in a group of modules. In accordance with at least someembodiments, the application (822) can be arranged to operate withprogram data (824) on an operating system (821).

The computing device (800) can have additional features orfunctionality, and additional interfaces to facilitate communicationsbetween the basic configuration (801) and any required devices andinterfaces.

System memory (820) is an example of computer storage media. Computerstorage media includes, but is not limited to, RAM, ROM, EEPROM, flashmemory or other memory technology, CD-ROM, digital versatile disks (DVD)or other optical storage, magnetic cassettes, magnetic tape, magneticdisk storage or other magnetic storage devices, or any other mediumwhich can be used to store the desired information and which can beaccessed by computing device 800. Any such computer storage media can bepart of the device (800).

The computing device (800) can be implemented as a portion of asmall-form factor portable (or mobile) electronic device such as a cellphone, a smartphone, a personal data assistant (PDA), a personal mediaplayer device, a tablet computer (tablet), a wireless web-watch device,a personal headset device, an application-specific device, or a hybriddevice that include any of the above functions. The computing device(800) can also be implemented as a personal computer including bothlaptop computer and non-laptop computer configurations.

The foregoing detailed description has set forth various embodiments ofthe devices and/or processes via the use of block diagrams, flowcharts,and/or examples. Insofar as such block diagrams, flowcharts, and/orexamples contain one or more functions and/or operations, it will beunderstood by those within the art that each function and/or operationwithin such block diagrams, flowcharts, or examples can be implemented,individually and/or collectively, by a wide range of hardware, software,firmware, or virtually any combination thereof. In accordance with atleast one embodiment, several portions of the subject matter describedherein may be implemented via Application Specific Integrated Circuits(ASICs), Field Programmable Gate Arrays (FPGAs), digital signalprocessors (DSPs), or other integrated formats. However, those skilledin the art will recognize that some aspects of the embodiments disclosedherein, in whole or in part, can be equivalently implemented inintegrated circuits, as one or more computer programs running on one ormore computers, as one or more programs running on one or moreprocessors, as firmware, or as virtually any combination thereof, andthat designing the circuitry and/or writing the code for the softwareand or firmware would be well within the skill of one of skill in theart in light of this disclosure.

In addition, those skilled in the art will appreciate that themechanisms of the subject matter described herein are capable of beingdistributed as a program product in a variety of forms, and that anillustrative embodiment of the subject matter described herein appliesregardless of the particular type of non-transitory signal bearingmedium used to actually carry out the distribution. Examples of anon-transitory signal bearing medium include, but are not limited to,the following: a recordable type medium such as a floppy disk, a harddisk drive, a Compact Disc (CD), a Digital Video Disk (DVD), a digitaltape, a computer memory, etc.; and a transmission type medium such as adigital and/or an analog communication medium (e.g., a fiber opticcable, a waveguide, a wired communications link, a wirelesscommunication link, etc.).

With respect to the use of substantially any plural and/or singularterms herein, those having skill in the art can translate from theplural to the singular and/or from the singular to the plural as isappropriate to the context and/or application. The varioussingular/plural permutations may be expressly set forth herein for sakeof clarity.

It should also be noted that in situations in which the systems andmethods described herein may collect personal information about users,or may make use of personal information, the users may be provided withan opportunity to control whether programs or features associated withthe systems and/or methods collect user information (e.g., informationabout a user's preferences). In addition, certain data may be treated inone or more ways before it is stored or used, so that personallyidentifiable information is removed. For example, a user's identity maybe treated so that no personally identifiable information can bedetermined for the user. Thus, the user may have control over howinformation is collected about the user and used by a server.

Thus, particular embodiments of the subject matter have been described.Other embodiments are within the scope of the following claims. In somecases, the actions recited in the claims can be performed in a differentorder and still achieve desirable results. In addition, the processesdepicted in the accompanying figures do not necessarily require theparticular order shown, or sequential order, to achieve desirableresults. In certain implementations, multitasking and parallelprocessing may be advantageous.

The invention claimed is:
 1. A computer-implemented method comprising:receiving, at a centralized processing device, a corresponding hotwordconfidence score from each of multiple media devices in communicationwith the centralized processing device via a network, each hotwordconfidence score indicating a likelihood that audio data correspondingto a first utterance of a user received by the corresponding mediadevice includes a particular, predefined hotword; determining, by thecentralized processing device, that two or more of the received hotwordconfidence scores satisfy a hotword score threshold; for each of the twoor more media devices having hotword confidence scores that satisfy thehotword score threshold, receiving, at the centralized processingdevice, second audio data from the corresponding media device, thesecond audio data recorded by the corresponding media device andincluding a user speech command; and generating, by the centralizedprocessing device, a request associated with the user speech commandbased on the second audio data received from each of the two or moremedia devices having hotword confidence scores that satisfy the hotwordscore threshold.
 2. The computer-implemented method of claim 1, furthercomprising, in response to receiving the corresponding hotwordconfidence score from each of the two or more media devices incommunication with the centralized processing device, instructing, bythe centralized processing device, each of the two or more media devicesto mute a corresponding loudspeaker.
 3. The computer-implemented methodof claim 1 further comprising, prior to receiving the second audio datafrom the corresponding media device, instructing, by the centralizedprocessing device, the corresponding media device to activate acorresponding microphone.
 4. The computer-implemented method of claim 1,wherein each of the multiple media devices are configured to: record thefirst audio data corresponding to the first user utterance; detect theparticular, predefined hotword in the first audio data using acorresponding hotword data module; compute the corresponding hotwordconfidence score indicating the likelihood that the first audio datarecorded by the corresponding media device includes the particular,predefined hotword; and transmit the corresponding hotword confidencescore over the network to the centralized processing device.
 5. Thecomputer-implemented method of claim 1, further comprising: transmittingthe request associated with the user speech command from the centralizedprocessing device to an external server; receiving, at the centralizedprocessing device, an audio response associated with the user speechcommand from the external server; and transmitting the audio response toat least one of the multiple media devices, the audio response whenreceived by the at least one media device causing the at least one mediadevice to play the audio response over a corresponding loudspeakerassociated with the at least one media device.
 6. Thecomputer-implemented method of claim 1, wherein generating the requestassociated with the user speech command based on the second audio datacomprises combining the second audio data received from each of the twoor more media devices having hotword confidence scores that satisfy thehotword score threshold to generate the request associated with the userspeech command.
 7. The computer-implemented method of claim 1, furthercomprising, for each of the two or more media devices having hotwordconfidence scores that satisfy the hotword score threshold, determining,by the centralized computing device, a word transcription and a wordsegment score for each word in the second audio data received from thecorresponding media device.
 8. A system comprising: one or morecomputers and one or more storage devices storing instructions that areoperable, when executed by the one or more computers, to cause the oneor more computers to perform operations comprising: receiving acorresponding hotword confidence score from each of multiple mediadevices in communication with the one or more computers via a network,each hotword confidence score indicating a likelihood that audio datacorresponding to a first utterance of a user received by thecorresponding media device includes a particular, predefined hotword;determining that two or more of the received hotword confidence scoressatisfy a hotword score threshold; for each of the two or more mediadevices having hotword confidence scores that satisfy the hotword scorethreshold, receiving second audio data from the corresponding mediadevice, the second audio data recorded by the corresponding media deviceand including a user speech command; and generating a request associatedwith the user speech command based on the second audio data receivedfrom each of the two or more media devices having hotword confidencescores that satisfy the hotword score threshold.
 9. The system of claim8, wherein the operations further comprise in response to receiving thecorresponding hotword confidence score from each of the two or moremedia devices, instructing each of the two or more media devices to mutea corresponding loudspeaker.
 10. The system of claim 8, wherein theoperations further comprise, prior to receiving the second audio datafrom the corresponding media device, instructing the corresponding mediadevice to activate a corresponding microphone.
 11. The system of claim8, wherein each of the multiple media devices are configured to: recordthe first audio data corresponding to the first user utterance; detectthe particular, predefined hotword in the first audio data using acorresponding hotword data module; compute the corresponding hotwordconfidence score indicating the likelihood that the first audio datarecorded by the corresponding media device includes the particular,predefined hotword; and transmit the corresponding hotword confidencescore over the network to the one or more computers.
 12. The system ofclaim 8, wherein the operations further comprise: transmitting therequest associated with the user speech command to an external server;receiving an audio response associated with the user speech command fromthe external server; and transmitting the audio response to at least oneof the multiple media devices, the audio response when received by theat least one media device causing the at least one media device to playthe audio response over a corresponding loudspeaker associated with theat least one media device.
 13. The system of claim 8, wherein generatingthe request associated with the user speech command based on the secondaudio data comprises combining the second audio data received from eachof the two or more media devices having hotword confidence scores thatsatisfy the hotword score threshold to generate the request associatedwith the user speech command.
 14. The system of claim 8, wherein theoperations further comprise, for each of the two or more media deviceshaving hotword confidence scores that satisfy the hotword scorethreshold, determining a word transcription and a word segment score foreach word in the second audio data received from the corresponding mediadevice.
 15. A non-transitory computer-readable medium storing softwarecomprising instructions executable by one or more computers which, uponsuch execution, cause the one or more computers to perform operationscomprising: receiving a corresponding hotword confidence score from eachof multiple media devices in communication with the one or morecomputers via a network, each hotword confidence score indicating alikelihood that audio data corresponding to a first utterance of a userreceived by the corresponding media device includes a particular,predefined hotword; determining that two or more of the received hotwordconfidence scores satisfy a hotword score threshold; for each of the twoor more media devices having hotword confidence scores that satisfy thehotword score threshold, receiving second audio data from thecorresponding media device, the second audio data recorded by thecorresponding media device and including a user speech command; andgenerating a request associated with the user speech command based onthe second audio data received from each of the two or more mediadevices having hotword confidence scores that satisfy the hotword scorethreshold.
 16. The computer-readable medium of claim 15, wherein theoperations further comprise, in response to receiving the correspondinghotword confidence score from each of the two or more media devices,instructing each of the two or more media devices to mute acorresponding loudspeaker.
 17. The computer-readable medium of claim 15,wherein the operations further comprise, prior to receiving the secondaudio data from the corresponding media device, instructing thecorresponding media device to activate a corresponding microphone. 18.The computer-readable medium of claim 15, wherein each of the multiplemedia devices are configured to: record the first audio datacorresponding to the first user utterance; detect the particular,predefined hotword in the first audio data using a corresponding hotworddata module; compute the corresponding hotword confidence scoreindicating the likelihood that the first audio data recorded by thecorresponding media device includes the particular, predefined hotword;and transmit the corresponding hotword confidence score over the networkto the one or more computers.
 19. The computer-readable medium of claim15, wherein the operations further comprise: transmitting the requestassociated with the user speech command to an external server; receivingan audio response associated with the user speech command from theexternal server; and transmitting the audio response to at least one ofthe multiple media devices, the audio response when received by the atleast one media device causing the at least one media device to play theaudio response over a corresponding loudspeaker associated with the atleast one media device.
 20. The computer-readable medium of claim 15,wherein generating the request associated with the user speech commandbased on the second audio data comprises combining the second audio datareceived from each of the two or more media devices having hotwordconfidence scores that satisfy the hotword score threshold to generatethe request associated with the user speech command.